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Abstract 

Intervention researchers in school settings often implement interventions that involve professional development 
to schoolteachers or other professional staff to implement school-based interventions. In terms of classroom 
interventions, teachers are the primary implementation agents; therefore, the fidelity of such interventions 
depends on teacher adoption and delivery. To understand and meet such fidelity challenges, we set to identify the 
contextual components and implementation mechanisms that contribute to the variation in teachers’ perspectives 
and methods of implementing an experimental evaluation of CareerStart; an innovative intervention designed to 
increase student school engagement and achievement by advancing teacher use of career relevant instruction in 
the middle grades. The results of this study suggest teachers perceive CareerStart in three interrelated ways: their 
perceptions of the program’s effectiveness on student outcomes; the degree to which the program is adaptable 
and compatible to their roles and responsibilities; and the effects of the school environment on their perceptions 
of the intervention. We conclude with a discussion of treatment fidelity effectiveness in school-based 
intervention programs. 

Keywords: treatment fidelity, professional development, intervention research, middle school, school 
engagement, school-based intervention, evidence-based practice, implementation 

1. Introduction 

Intervention researchers working in school settings recognize that teachers and educational professionals can 
play vital roles in the successful implementation of school- and classroom-based interventions. In particular, 
layering interventions on existing structural and social environments such as found in schools present unique 
challenges to researchers. Further, difficulties arise from the complexities associated with the school as: 1) a 
“host” research or practice setting for intervention researchers who are not teachers, 2) a learning environment 
for youth, as well as 3) a workplace for teachers and staff (Orthner, Cook, Sabah & Rosenfeld, 2006; Woolley, 
2007). Successful implementation of interventions designed to change, for example, student engagement in 
school, frequently involve changing selected aspects of work-related interactions and school or classroom 
contextual factors. This study explores how such complexities can compromise the fidelity of school-based 
interventions, which presents significant challenges to researchers who seek to design and implement 
interventions in school settings. 

The potential value of understanding this complex layering effect on treatment fidelity presents significant 
implications for the growing volume of research focused on the impact of teaching strategies, the teacher-student 
relationship, and the learning climate in schools. Perry, Liu, and Pabian (2010) found that teacher support has a 
positive effect on student engagement, with higher levels of exposure predicting improved student academic 
performance. Likewise, Woolley and Grogan-Kaylor (2006), in a study including measures across three 
influential settings for students—home, neighborhood, and school—found teacher support to be the most 
influential environmental factor in predicting a student’s sense of school coherence and academic performance. 
Orthner, Cook, Sabah and Rosenfeld (2006) reported in a cross-national study that enhancing the organizational 
learning environment of the classroom benefitted student behavior in the classroom and improved teacher morale 
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and performance. 

1.1 Teacher Factors in Achievement 

A positive teacher-student relationship also helps facilitate student learning. In addition to fostering positive 
relationships, teachers have implemented pedagogical changes to facilitate student motivation and engagement in 
the learning process, especially through strategies that utilize contextual relevance in classroom lessons (Keller, 
1983; Means, Jonasson, & Dwyer, 1997). These changes can include strategies to increase the relevancy of 
instruction related to contemporary circumstances of students as well as future college and career opportunities. 
Relevancy of instruction improves students’ sense of purpose, connection to their learning, and beliefs that their 
personal goals are attainable (Orthner, Akos, Rose, Jones-Sanpei, Mercado, Woolley, 2010). In order to increase 
relevant instruction school systems often encourage teachers to apply career exploration activities in their 
classrooms. This approach has been shown to have a positive impact on student engagement (Kenny & Bledsoe, 
2005). 

Because teachers implement instructional interventions in their classrooms, implementation fidelity hinges on 
teacher adoption. Treatment fidelity is the capacity, ability, and extent to which teachers implement a program as 
it was intended (Fraser, Richman, Galinsky, & Day, 2009; Tucker & Blythe, 2008). In the case of CareerStart, we 
operationalize fidelity as extent of use of the prepackaged CareerStart lessons provided by the intervention to the 
teachers (while acknowledging that the use of career examples in pedagogy is not an uncommon strategy). 
Threats to fidelity include factors that lead to the diminution of consistent delivery of the core elements of an 
intervention, in the case of CareerStart, that means the use of the curriculum linked lessons. Still, there are many 
factors that can threaten treatment fidelity; therefore, researchers seek high levels of standardization through the 
development of strategies such as program manuals, professional development training, and requiring 
individuals implementing an intervention to demonstrate competence through an evaluation or certification in the 
delivery of an intervention (Tyler & Blythe, 2008). Despite such efforts, researchers and teachers face a myriad 
of challenges to high fidelity service delivery. 

In this study, we investigate teacher factors, which influenced CareerStart treatment fidelity by conducting focus 
groups with randomly selected teachers in several schools implementing CareerStart in 2006-07 and 2007-08. 
Our findings suggest that teachers’ perceptions of the program, its compatibility to their roles and responsibilities, 
and the influence of the school environment have implications for education researchers working with teachers 
to implement changes in classrooms. We link our findings to the literature and propose that what we have found 
lends support to what other researchers have found regarding implementation fidelity with teachers and has 
implications for researchers and practitioners working in educational settings (Smith, Daunic, & Taylor, 2007; 
Wehby, Maggin, Partin, & Robertson, 2011; Zvoch, 2009). 

1.2 The Intervention: CareerStart 

Developed by an interdisciplinary research team in partnership with educators in both a university school of 
social work and school of education, and a school district, CareerStart is a curriculum-based intervention 
program designed to improve student engagement in school and prevent progressive student disengagement 
across the middle grades. CareerStart lessons were developed and peer reviewed by teachers under a rubric that 
linked the new lessons to current priority content in core middle school subject areas. After review by curriculum 
specialists, the lesson plans and resources were made available on line for 6 th , 7 th and 8 th grade core subject 
teachers. 

1.2.1 Student Engagement 

Two aspects of student engagement are targeted: 1) emotional engagement, which is characterized by a student 
valuing school, seeing school as important to his or her future, and feeling he or she can succeed at school, which 
theoretically leads to 2) behavioral engagement, which includes attending school, following the rules, and 
completing assignments (Fredricks, Blumenfeld, & Paris, 2004). Substantial evidence indicates student 
emotional and behavioral engagement is positively associated with academic performance, persistence, and 
completion (Finn, 1993; Fredricks, Blumenfeld, & Paris, 2004; Perry, 2008; Woolley & Bowen, 2007). It is 
alarming, therefore, that current research indicates that student engagement typically declines across the middle 
and high school years for all students, including those at risk for school failure (Anderman, & Midgely, 1997; 
Martin, 2009; Wigfield & Tonks, 2002). 

CareerStart was designed to advance student engagement by improving students’ understanding of the relevance 
of their school coursework to jobs and careers that use the skills learned in the classroom, thereby influencing 
how students perceive their possible futures. It is an innovative program in a number of ways. First, it strives to 
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reduce high school drop out by intervening at the middle school level when student disengagement from school 
experiences its greatest. In contrast, dropout programs are most often implemented in high schools, arguably 
after trajectories toward failure and dropout are already established. Second, rather than the intervention being 
implemented in a separate course or module, CareerStart intervenes directly within the core academic 
coursework of middle grade students: math, language arts, science and social studies courses. Third, teachers are 
the CareerStart practitioners, trained and provided with well-developed lessons and instructional materials by the 
implementation team. Finally, CareerStart is a universal strategy in that all middle school teachers across sixth, 
seventh, and eighth grades implement the lessons to all students, allowing each student to experience the 
intervention throughout the middle grades. Figure 1 depicts these innovative characteristics as well as an 
overview of the theoretical foundation informing intervention development and implementation. 

Implementing CareerStart in middle schools involves wrestling with the layering effects and complexity of the 
school environment as both work and learning environments, as well as non-educators coming into a school and 
training teachers with the goal to change what they do in their classrooms, both of which can exacerbate 
challenges in achieving high implementation fidelity. This layering and complexity can be seen in the 
CareerStart theory of change logic model in Figure 1. In order to change student outcomes, the intervention 
involves changing teaching style and strategies, which is dependent on the organizational changes implemented 
by school principals. 



Figure 1. CareerStart Theory of Change Model 


CareerStart training, resources and technical assistance equip teachers with example on-line lessons, and 
career-relevant instmctional tools that help them relate classroom material to the ways in which adults in jobs 
and careers might apply such information. Research findings indicate CareerStart increases student academic 
achievement and reduces school disengagement and student misbehavior (Orthner, Jones-Sanpei, Rose, Mercado, 
Akos, 2010). The approaches teachers use to implement CareerStart are attributable to diverse factors such as 
their levels of experience (Rose, Woolley, Akos, Orthner, Mercado, & Jones-Sanpei, 2012), preferred teaching 
styles, acceptance of the intervention model, and the diverse characteristics and needs of their students. 

1.2.2 Prior Research on CareerStart 

Previous research emerging from a randomized control trial previously found positive effects from CareerStart. 
Those effects demonstrated that CareerStart: 1) effectively advances teacher use of career relevant material in 
their lessons (Rose et al., 2012); 2) improves the value students perceive in their education as important to their 
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futures (Orthner, Jones-Sanpei, Akos, & Rose, 2013); and 3) improves student achievement on math 
end-of-grade tests (Orthner, Rose, & Mercado, 2009; Woolley et al., in review). Further, findings with respect to 
increased teacher use of CareerStart materials illustrated that: a) on average teachers successfully implemented 
the treatment in their lessons; b) there were variations at the teacher level in this implementation; and c) closer 
examination of the teacher-level sources of this variation in fidelity are central to understanding how it can be 
improved. 

1.2.3 Teacher Fidelity 

Quantitative data underscores this variation. Table 1 shows teacher fidelity as measured by the proportion of 
lessons delivered each year between 2006-07 and 2008-09. For example, the proportion of lessons implemented 
in 2008-09 (64%) exceeded the proportion of lessons delivered in 2006-07. 


Table 1. Teacher Fidelity by Proportion of Lessons Delivered 


Year 

Mean 

Standard 

Deviation 

Skewness 

Kurtosis 

2006-07 

0.58 

0.26 

0.06 

-1.29 

2007-08 

0.54 

0.24 

-0.08 

-0.75 

2008-09 

0.64 

0.24 

-0.33 

-0.65 


The distributions of lessons taught for each cohort are illustrated in Figures 2, 3, and 4. Across the three years, 
the percent of teachers who taught at least five lessons increased. These graphs illustrate how treatment fidelity 
changes over time, providing important information to understand the implementation process in the program. 



Percent of Lessons (of 10) 


Figure 2. Teacher Fidelity in Year One 



Percent of Lessons (of 10) 


Figure 3. Teacher Fidelity in Year Two 
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Percent of Lessons (of 10) 


Figure 4. Teacher Fidelity in Year Three 


1.3 Treatment Fidelity 

Treatment fidelity is the extent to which program implementation consistently represents the intended model 
(Fraser, Richman, Galinsky, & Day, 2009). Treatment fidelity in school-based interventions is monitored using 
both qualitative and quantitative methods to ensure students are receiving intended quality services (O'Hare, 
2005). Examples of qualitative methods include case analyses, direct observational studies, or process 
assessments (O'Hare, 2005); with examples of quantitative methods include standardized questionnaires or 
surveys. Measuring the implementation process also gauges internal validity, that is, the extent to which the level 
of confidence researchers can assert that observable differences in outcomes—or lack thereof—were due to the 
intervention and not extraneous factors (Tucker & Blythe, 2008). 

However, too few research studies in school settings assess treatment fidelity. In a review of fifty-five 
school-based intervention studies identified by computer searches of the Education Resources Information 
Center (an abstract database commonly known as ERIC) and PsycINFO (an abstract database maintained by the 
American Psychological Association), 27 percent of published articles between 1975 and 2002 discussed 
treatment fidelity (Mooney, Epstein, Reid, & Nelson, 2003). Although some studies that assessed fidelity also 
implemented strategies to enhance treatment fidelity, there appears to be sparse research on the use of treatment 
fidelity findings to improve treatment fidelity (Tucker and Blythe, 2008). These finding underscore the need for 
more attention to treatment fidelity in educational research as a necessary component for establishing 
evidence-based practices in education (McLeod, & Southam-Gerow 2009; Mowbray, Holder, Teague, & Bybee, 
2003; Smith, Daunic, Taylor, 2007). Below, we describe two common components of treatment fidelity 
monitored in treatment implementations. Such data can also be effective for understanding participants’ 
implementation methods, strengthening treatment fidelity protocols, and informing subsequent analyses and 
implementations. 

1.3.1 Contextual Factors 

In evaluating treatment fidelity, the implementation context often helps explain the degree to which 
implementation processes have an effect on treatment outcomes. In a meta-analysis of 81 studies, Durlak and 
DuPre (2008) identified 23 contextual factors that influenced implementation and situated those factors in three 
contextual categories: 1) community factors, 2) provider characteristics, and 3) intervention characteristics 
(Durlak & DuPre, 2008). Community factors affect an intervention by means of such dynamics as politics, 
funding, or policies. For example, the successful implementation of a multi-school intervention program may be 
conditional on school officials’ ability to petition for funding of the intervention through political support. School 
officials include the superintendent, principals, teachers and other school administrators, who are in the position 
to facilitate or delay implementation. 

The second contextual category includes teachers’ characteristics including perceptions of the intervention, 
beliefs about the need for the intervention, self-efficacy, teaching experience, and skill competence. For example, 
experienced teachers, already having had years of teaching experience and feeling skilled with numerous 
teaching strategies, may be more reluctant to change their established teaching practices, thus affecting treatment 
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adherence. However, compared to seasoned teachers, novice teachers may respond favorably to an intervention 
they assume may help strengthen their skills and add to their repertoire of teaching practices (Stein et ah, 2008). 
Teachers with positive perceptions of an intervention may demonstrate confidence, consequently affecting their 
level of adherence and contribution to overall fidelity compared to teachers with low opinion of the intervention 
(Durlak & DuPre, 2008). This suggests researchers’ understanding of the variation of teachers’ characteristics is 
an important dimension of prior intervention implementation. 

The last contextual category is intervention characteristics, which refers to the ease in which teachers can adapt 
the intervention to their needs and the extent to which the intervention is compatible with their roles, 
responsibilities, priorities, and the mission of the organization. An intervention that is adaptable to teachers 
suggests treatment fidelity is feasible when the components of the intervention complement their competencies. 
Overall, Durlak and DuPre (2008) suggest that treatment fidelity may be attainable and sustained when 
intervention processes fit the organizational context, in that the components align with both teachers’ 
characteristics and the established systems and structure in which teachers work. 

1.3.2 Social Validity 

Social validity refers to both teachers’ level of acceptance of an intervention and perceptions of intervention 
effectiveness in the classroom (Carter & Pesko, 2008). However, educators’ acceptance of school-based 
interventions is a sparsely researched area (Carter & Pesko, 2008). The relationship between social validity and 
implementation fidelity is an extension of Wolf’s (1978) research on the importance of seeking feedback from 
those who are in the role of implementing a treatment in order to understand a program’s social significance. 
Wolf (1978) explained that changes in behavior assessed by an objective measure—for example direct 
observation utilizing a reliable instrument—may contradict for example, a teacher’s report of the effectiveness of 
the program. For example, a child in a treatment program may show slight improvement in behavior, but 
feedback from the teacher may indicate significantly higher improvement. In this case, the difference between 
teacher feedback and the objective behavior score underlines Wolf’s contention that intervention effectiveness is 
contingent on the social validity of the teacher. 

Whereas Wolf discusses parents’ social validity, here the connotation applies to the teacher, as the direct 
consumer, although not the eventual target of the intervention. Following this context, social validity refers to the 
teacher’s subjective perception of the acceptability and feasibility of the procedures, and the value of the effects 
of the intervention (Wolf, 1978). For example, the application of intervention strategies (e.g., low adherence to a 
particular lesson) likely implemented by teachers’ in their teaching practices may emerge from their perceptions 
of the intervention. 

For teachers, one aspect of intervention feasibility is the extent to which the proposed intervention complements 
existing teaching practices (Carter & Pesko, 2008). Rather than competing for teachers limited time, energy, and 
resources, or expecting them to exceed their roles and responsibilities, intervention practices that match 
classroom dynamics positively influence teachers’ decisions to accept and adhere to a new intervention protocol 
(Carter & Pesko, 2008). For example, in an evaluation of a school-based violence prevention program, Biggs and 
colleagues (2008) examined the social validity teachers placed on a particular intervention and its effect on 
teachers’ decisions to implement the program. The researchers assumed social validity to be a function of 
perceived fit of the intervention with existing teaching practices. In measuring social validity, these researchers 
assessed: a) teachers’ acceptability of the program; b) the degree to which their attitudes and belief systems 
aligned with the purpose of the intervention; and c) their perceptions of the effectiveness of the intervention 
(Biggs et ah, 2008). Findings suggested teachers’ level of new program acceptability and adherence aligned with 
the ease to which teachers felt that program components could be incorporated into their classroom routines, as 
opposed to representing discrete new classroom activities. Further, teachers’ perceptions of the acceptability and 
usefulness of an intervention were significantly associated with program fidelity (Biggs et ah, 2008). 
Interestingly, teachers’ reports of the utility of the intervention increased significantly over time. These 
assessments also provided opportunities for teachers to comment on the intervention. Although most comments 
were positive, teachers’ expressed concerns regarding the limited time allocated for collecting data for the 
intervention, and the perceived unnecessary requirements to change some of their teaching practices. 

In a report of a similar research endeavor, Martens and McIntyre (2009) discussed the importance of evaluating 
educators’ acceptability of an intervention. Acceptability here refers to the match between the characteristics of 
the intervention and teachers’ own perceptions of its appropriateness and value (Martens & McIntyre, 2009). The 
authors contend assessing practitioners’ willingness to implement a program is as important as demonstrating 
treatment effectiveness. They explain practitioners may be more likely to respond positively to the intervention 
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(e.g., increased levels of program commitment, higher rate of implementation fidelity, sustainability of program 
protocol over time) when educators’ perceive an intervention as useful. Such research findings reinforce the 
layering effect and two level nature—teachers in their work environment and students in their learning 
environment—of the school as a setting for intervention research discussed above. However, Martens and 
McIntyre (2009) contend that acceptability does not necessarily yield consistent and sustainable implementation 
of intervention protocols by teachers. To remedy declining fidelity among teachers, continuous support and 
reinforcement, monitoring, and feedback are suggested methods for ensuring implementation integrity over time 
(Martens & McIntyre, 2009). 

Assessing teacher acceptability helps gauge the likely success of the implementation of an intervention. 
According to Chafouleas, Briesch, Riley-Tillman, and McCoach (2009), practitioners’ responses to an 
intervention as “appropriate, fair, and reasonable” can be assessed, especially when relevant, appropriate 
modifications to the intervention are undertaken to strengthen teacher acceptability and treatment effectiveness 
(Chafouleas, et ah, 2009). However, Chafouleas and colleagues (2009) also note that attaining high levels of 
acceptability alone does not necessarily result in teachers’ effective use of the intervention program or its 
components. For example, in a multi-component drug prevention program evaluated by Stead and colleagues 
(2007), teachers’ maintained an acceptable level of fidelity despite not understanding the rationale for conducting 
specific practices that were outlined in the intervention model. Qualitative interviews indicated that teachers 
exercised professional discretion and modified or excluded lesson components as they grew familiar with the 
intervention. 

The contention that training and other forms of professional skill development are essential for ensuring fidelity 
has been widely researched, concluding that there is a positive correlation between training and high levels of 
fidelity (Stead et ah, 2007). For example, Stead and colleagues (2007) concluded that time constraints in teachers’ 
ability to deliver quality lessons, teachers’ concerns about classroom disruptions during interactive assignments, 
and teachers’ recognition of their limited content expertise were factors affecting their responses to the 
effectiveness of a new intervention (Sobeck, 2006). Further, program complexity appears to be associated with 
implementation, in that teachers may be less likely to implement complex interventions with high fidelity 
(Sobeck, 2006). In view of the challenges of maintaining treatment fidelity, in this study we explore the function 
of contextual factors and social validity on treatment fidelity of an intervention built around professional 
development training of middle grades teachers. 

2. Method 

2.1 Sample 

In November 2008, qualitative data were collected from teachers focused on the implementation of the 
CareerStart program in the teachers’ classrooms. We randomly selected 30 seventh grade teachers for four focus 
groups in four different schools. The teachers provided feedback about their use of CareerStart lessons, both 
positive and negative, as well as responses that they heard from their students about career examples being 
offered in the classroom. Two trained researchers led all group discussions. Sessions were tape-recorded, 
transcribed, and narratives of the conversations entered into Atlas-ti, version 5, a qualitative analytic software 
package (Atlas Ti, v5, 2008). 

2.2 Data Analysis Strategy’ 

Grounded theory informed the analysis of the data collected in the current study. Grounded theory is a recursive 
inductive process used to analyze and classify text data, followed by data hypotheses and theories that emerge 
from the analytic process (Padgett, 1998). Atlas.ti software facilitated open coding and a hierarchical 
classification procedure for themes emerging from the data. Open codings were used to identify preliminary 
codes for the narrative data, to provide a process for making memos and notes related to the coding, and to 
classify those codes and notes into themes. Described as a “cognitive act of assigning a code” (Lofland & 
Lofland, 1995), in this process data coders extract patterns related to events, key words, processes, and 
characters from the data, and then categorize them (Coffey, 2007). The grounded theories developed from this 
analysis were then compared to the literature previously described to triangulate these findings with what is 
already known about layering interventions in K-12 education settings. 

2.3 Data Analysis 

Members of the CareerStart research team initially read all transcriptions twice at two different time points for 
literal inteipretation and application of low and high inference coding (Anastas, 2004). These inference codings 
subsequently tested to see if they verified relationships between themes. Coffey and Atkinson (1996) report that 
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this procedure allows each quotation to derive two meanings, the immediate context to which it belongs, and the 
thematic information it provides. Subsequently, thematic categories were developed and integrated through these 
analyses (Weiss, 1994). Researchers documented coding procedures using memos. This technique allowed the 
research team to describe coding procedures. By recording the researchers’ decision-making processes, revisions 
and discrepancies among raters were resolved. 

3. Findings 

The current findings illuminate the effects of contextual factors on treatment fidelity as well as the effects of 
teachers’ perceptions of the social validity of the intervention on treatment fidelity. Three themes emerged from 
analytic process: 1) program factors, 2) individual factors, and 3) school community factors. 

3.1 Program Factors 

Programmatic factors consisted of two subcategories: adaptability of the intervention for implementation, and 
the compatibility of the intervention to teachers’ role and responsibilities in the classroom. Teachers’ level of 
adaptability varied between implementing the intervention as intended and modifying the intervention to meet 
contextual dynamics in the classroom. Teachers indicated two factors contributing to the variation of adaptability. 
In the first variation of adaptability, teachers discussed the time constraints they faced when implementing a 
CareerStart lesson. For example, a 7th grade teacher indicated that lessons had 

...too much content, in particularly in the social studies and math. There are too many things to do 
in a lesson plan. In one [lesson], we have to take 60 minutes.. .every lesson for CareerStart has been 
drastically modified. And for 45-minutes class period that forces down a lot. And the overwhelming 
factor to a minimum and stick to a few basic concepts per lesson. 

Similarly, a language arts teacher commented, 

I am teaching writing, reading and vocabulary and CareerStart so my time is limited and when you 
are working with children in levels 1 and 2, when we want them in 4s, the time factor is there. When 
that lesson is good might give them a better basis to make decisions about their careers. I just didn’t 
have time. 

Many teachers’ characterized lessons as having too much content, and one teacher described the successful 
implementation of a lesson as dependent on inconsistent available classroom time. For example, the 
implementation of math activities varied during the week, in that math teachers implemented activities on some 
days more than other days. Teachers explained that partial implementation was appropriate as long as students 
were exposed to the assigned math activity. 

The second variation of adaptability was teachers’ ability to modify lessons for their students. Teachers stated 
that although lessons were partially completed, they had the option of returning to a lesson to either review a 
concept, or continue that lesson later. As one teacher stated, “we can break them up in quarters.” Teachers also 
explained that adaptability was in part the fit between a CareerStart lesson and a required subject unit. Although 
one teacher asserted a high level of adaptability of the lessons: 

[The lessons] are more adaptable to various learners. Especially for our population we can illuminate 
this part, it is very adaptable, and we have that adaptability to do a part that gets the most of it or 
eliminate this part. So it is very adaptable. 

Flowever, other teachers expressed that sometimes matching a CareerStart lesson to a teaching unit was difficult, 
or that the lesson was not easy to implement across varying student-learning levels. Although some teachers 
stated that modifying the lessons was difficult, one teacher suggested success in personalizing a CareerStart 
lesson: 

I generally find it more successful if you personalize it to what you are doing in your classroom, like 
instead of using a generic job application you adapt it to what you are doing. 

Along similar lines, a group of 7 th grade math teachers used CareerStart math lessons as “a good jumping off 
point for discussion and added more content to classroom discussions”. Teachers commented positively on their 
ability to modify lessons; however, one teacher suggested problems with modification, in particular, partitioning 
the lesson: 

This is not a day’s lesson for us. If [students] reading abilities are limited they see all those words 
and they shut down. So, I do bits and pieces but then the problem with that is that the sequence is 
lost and absentees [miss parts of the lesson]. 
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In general, teachers agreed that the lessons were flexible and easy to adapt to various classroom academic levels, 
but found it challenging to ensure students learned the material efficiently. 

Along with issues of adaptability, the theme of compatibility between the intervention and teachers’ roles and 
responsibilities also emerged from the data. As previously indicated, teachers were responsible for managing 
their time efficiently to ensure the intervention was implemented as intended. As one teacher indicated, 
implementing lessons in classroom periods lasting 45-minutes did not match established teaching schedules. An 
example of a conflict between the compatibility of the intervention with teachers’ responsibilities was apparent 
when a teacher explained that he taught an advanced class and the intervention did not align with his role: 

... [the lessons are not] that interesting to kids. They teach everyday things but I want my kids to think 
higher, you are talking about college degrees these kids could have. The careers they use in there are not 
very interesting. 

Teachers’ comments concerning the compatibility of the intervention were also based on the characteristics of 
their students. For example, one teacher requested that lessons “incorporate different levels that would give us 
some idea of how to differentiate the lesson” for students’ diverse learning needs. In this case, the teacher was 
indicating that the lessons did not meet the direct needs of students who were having difficulty understanding or 
completing the objectives set by the lessons. Another teacher pointed out the lessons were not compatible to his 
students’ perceptions of interesting jobs. This alludes to the need to ensure that lessons are compatible with 
teachers’ perceptions of what they know or assume to be interesting to their students. In addition, a social studies 
teacher commented that the lessons were similar to existing teaching practices in her subject. Although this 
comment suggests compatibility between her tasks and the intervention, the teacher is highlighting that the 
intervention is not significantly different from her previous responsibilities. 

Another issue concerning compatibility was about the timing of introducing CareerStart to 7 th grade students. 
One teacher reported there to be a discrepancy in introducing career-related concepts in middle school rather 
than in high school. The teacher believed that career-related instruction would be more effective in high school 
when students are making career-related decisions about their coursework and future education. This comment 
suggests that there is a divergence in opinion between the objectives of the intervention and practitioners’ 
compatibility with its objectives. By understanding teachers’ perspectives of an intervention, specifically how 
adaptable and compatible the intervention is to their teaching practices and their perceptions of what students 
need and want, program developers may strengthen programmatic factors to better meet the objectives of the 
intervention. Another option may be that program developers purposively educate teachers about the conceptual 
basis of an intervention so they understand how an intervention may meet students’ needs and wants. 

3.2 Individual Factors 

Several individual factors associated with implementation were identified in the data. The manner in which 
educators’ perceived the intervention varied in two ways: 1) perceptions related to the need for the intervention, 
and 2) the intervention’s effectiveness among students. Teachers had mixed perceptions of the intervention. 
Some teachers indicated that the lessons seemed disjointed, arbitrary, and indistinguishable from existing units, 
while other teachers reported that the lessons seemed relevant, and well planned. Many novice teachers 
perceived the lessons as supportive and a useful resource in facilitating lesson completion. They also indicated 
that CareerStart lessons allowed them, as one teacher stated, “to focus on a subject matter [in a] different way.” 
Furthermore, a teacher commented, “it is a good jumping off point for discussion as we plan.” Some teachers 
also perceived students’ responses to lessons positively. One teacher stated, “I remember that I used one [lesson] 
and it worked. I was really surprised.” After implementing an entire unit, one teacher reported, “The kids reacted 
positively. I noticed that the following day, after they had the career start lesson, they were more serious about 
their learning, which I think is very positive.” 

Likewise, another teacher said she perceived the lessons brought relevancy to her classroom. In contrast, the 
teacher of an advanced course perceived his students to be uninterested in the lessons because they were not 
challenging enough. A second teacher with a similar opinion commented: 

1 think that if we really wanted to get these kids interested in these careers, [then we need to] give them 
something really interesting that will hold their mind. 

She indicated that she perceived the focus of the intervention to be on traditional rather than emerging jobs. 
Contritely, another teacher welcomed the usefulness of bringing career content to the classroom: 

A bit of application is what you want. I think it’s [lessons] are useful. I think the idea is good in 7 th 
grade. I tell my kids they have to think about college so you have to start thinking about careers. As far 
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as the program, trying to implement that is a great idea. 

As displayed in this dialogue, teachers expressed mixed feelings for the intervention. While some teachers 
already incorporated career-relevancy into their teaching practices, other teachers experienced the program as a 
new and interesting resource for their students. Teachers’ perceptions for the need and usefulness of an 
instructional strategy influences implementation fidelity. 

3.3 School Community Factors 

School community factors refer to factors that affect the implementation process of an intervention such as 
politics, funding, and broader state or federal policies (Durlak & DuPre, 2008). Teachers provided comments 
about external factors affecting their perceptions about implementing CareerStart. For example, one teacher 
voiced her opinion by expressing her discontent with how her school administrators introduced the intervention 
as a required teaching practice method rather than a resource to use with her students. Further, although this 
program was introduced as a “school system directive” a second teacher indicated that it was irrelevant because: 

In math the way I teach, it is automatic that I am going to talk about careers. I am already doing this. 

School policies also mitigated how teachers implemented the intervention. In one case, a teacher asked 
permission to use a similar lesson not related to CareerStart. The school administration indicated that the new 
lesson had to be in conjunction with the existing language arts CareerStart lesson. Similarly, another teacher 
commented that. 

Lessons are strictly tied into the standard course of study and if you go beyond that, you have 
enrichment units. [Then] you are sort of on your own. 

Community factors, specifically the influence of school administrators on teachers’ perceptions of an 
intervention, are an area of research that needs further exploration. In this study, teachers’ perceptions of the 
intervention appear affected by the school environment in which they work. 

4. Implications and Discussion 

We conducted a qualitative investigation of factors that affected teacher fidelity of implementation of an 
instructional relevance intervention. Teacher implementation of CareerStart, developed by an interdisciplinary 
team of researchers and educators, was shown to have varying levels of fidelity as measured by teachers’ reports 
of lessons used. Our findings have implications for further study and development of CareerStart and more 
broadly for the literature on fidelity of implementation in school settings. 

4.1 CareerStart Fidelity’ Implications 

We learned many valuable lessons in the design and implementation of CareerStart, a school-based intervention 
research project. Most fundamentally, we learned the lesson that all school-based researchers must learn, that 
working with, listening to, and valuing the experience, input, and roles of teachers in the implementation of new 
interventions delivered in the classroom is as important as the design, theoretical underpinnings, and content of 
that intervention. More valuable, however, are the details and specific lessons learned in terms of how to engage 
teachers and at what points in the intervention development and implementation process to maximize treatment 
adherence and therefore fidelity. These lessons include: 

a) Early in the design process collect systematic data—observations and teacher interviews or focus 
groups—about the current instructional process and procedures that are being used in the classrooms that will be 
the target of the interventions; 

b) At a midpoint in the design of an intervention, it is important to seek feedback from teachers about 
how the intervention can better fit and be integrated into their standard instructional approach and strategies: 

c) Pilot test training materials and treatment manuals and get feedback from teachers about the clarity, 
ease of use, and effectiveness of those materials; 

d) Once program implementation begins in a systematic and ongoing manner, monitor adherence and 
fidelity by collecting both qualitative and quantitative data; and finally, 

e) When problems in adherence or fidelity are found, immediately get more data from teachers and seek 
to understand the sources of their challenge; then make program changes to address the issues they raise. 

As part of this evaluation project, CareerStart was originally introduced to the first wave of sixth grade students 
in 2005-2006 by their teachers, who had been trained to use the fully developed lessons. In order to strengthen 
the fidelity of teacher implementation in the second year, which was 2006-2007, several measures were 
undertaken. A teacher-coach who had initially implemented CareerStart assisted in modifying the program to 
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make it more easily adopted by other teachers. Second, modifications were made to the lessons informed by 
feedback from the teachers. Third, a system of active accountability was developed and put into practice, which 
produced quarterly reports of lessons taught at the teacher and school levels, including feedback on lesson 
quality and applicability to their students (Orthner, et al., 2010). Quantitative analysis of data collected during 
and after the first year has since confirmed CareerStart did not demonstrate statistically significant gains in 
student achievement. After consulting with teachers during and following the first year of implementation, the 
program was restructured to consist of 10 shorter and manageable “mini-lessons” per core subject. New 
personnel joined the implementation team, including a teacher who participated in the first implementation year 
of CareerStart. As an on-site coordinator for CareerStart, she served in this role until the end of the evaluation 
project. Based on the feedback received from the qualitative and quantitative data, the substantially updated 
program has demonstrated significant positive effects on student engagement and achievement. 

4.2 School-Based Research Fidelity Implications 

In any teacher implemented intervention study, understanding the nature of usual treatment or services is 
essential. Assessing the usual treatment prior to an instructional intervention implementation and evaluation is 
uncommon. Measuring routine teacher behavior as an element of the general intervention work is costly. Still, 
such procedures are important because understanding similarities and differences between pre-intervention 
teaching processes and new intervention processes may help educational program developers and researchers 
better understand the degree to which a new instructional intervention: a) overlaps with current services, b) 
exceeds or deviates from practitioners’ roles and responsibilities, or c) undermines targeted teacher knowledge, 
skills, or supports. 

Given that instructional strategies are implanted in the classroom, measuring teacher level of fidelity is helpful to 
ensure that outcomes are robust and valid in such intervention research (Tucker & Blythe, 2008). As previously 
indicated, there are several methods to evaluate treatment fidelity including randomized observational checks, 
videotaped intervention sessions, audio taped sessions, process assessment, standardized evaluation forms, and 
post-session assessment. Such methods help evaluators and researchers organize and categorize barriers to 
teacher level fidelity, identify the effectiveness of program components, incorporate practitioners’ suggestions, 
and distinguish variations found in program outcomes. 

4.3 Conclusion 

This study contributes to the literature on treatment fidelity in school-based intervention research. In the current 
study, we used qualitative methods to examine teacher factors influencing fidelity in the implementation of 
CareerStart. The current findings suggest teachers perceived the early version of CareerStart in three interrelated 
ways: 1) their perceptions of the need for the program and its effectiveness on student outcomes; 2) the degree to 
which the program is adaptable and compatible to their roles and responsibilities; and 3) the affects of the school 
environment on their perceptions of the intervention. 

The current findings show potential for informing researchers of the merit and efficacy of the intervention on 
student outcomes. Thus, research of treatment fidelity has the potential to advance the quality of evidence-based 
program development and research. Further, growing demands to demonstrate the efficacy and scalability of 
school reform programs should drive the need for fidelity evaluations to identify variations of program 
effectiveness across different populations. For that reason, research on treatment fidelity adds value to existing 
services, especially when it helps practitioners’ enhance their knowledge and skills, thus better standardizing 
treatment delivery. 

In the main, intervention research strives to establish causal inferences between treatments and outcomes 
(Borrelli et al., 2005). Without careful attention to fidelity, researchers may inaccurately over- or under-estimate 
the effects of an intervention (Fraser et al., 2009). Further, when treatment fidelity is attended to and improved, 
measurement validity is advanced, statistical power increased, and face validity of treatment effects strengthened 
(Borrelli et al., 2005; Summerfelt, 2003). In conclusion, the central goal of the current research was to contribute 
to our understanding of treatment fidelity in school-based intervention research. The current study met our goal 
in terms of advancing the fidelity of future implementations of CareerStart and we hope other school-based 
researchers likewise find that the current findings inform their efforts to advance school intervention fidelity. 
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